🕷️️ Job Radar • SCRAPING

freelancer.com 🟡 2026-05-19

🔹 Automate classified website data scraping for product information into an existing SQL database
👤 Client: 🇮🇳 indore, India Member since 2020-05-22
💰 Price: $68 Average bid
🚩 Problem: Collecting structured product details from a classified site requires efficient automation, accurate parsing, and reliable database integration.
📦 Existing: SQL database, Python environment

Specifications:

[Target] Classified website URLs
[Method] Python with Scrapy for web scraping; SQLAlchemy for database interaction
[UI/UX] N/A
[Stack] Python, Scrapy, BeautifulSoup, SQLAlchemy, PostgreSQL (or MySQL)
[Security] Secure database connection details stored externally; logging errors and progress
[Format] JSON for data structure, SQL dump for schema verification

Workflow:

1. Define the target URLs and set up a Scrapy project with custom spiders to handle pagination.
2. Implement functions to extract product titles, descriptions, prices, availability, and image URLs using BeautifulSoup or similar libraries.
3. Validate extracted data for accuracy and consistency before processing.
4. Use SQLAlchemy ORM to connect to the existing SQL database and implement upsert logic to avoid duplicates.
5. Download images and store them in a designated directory or reference their URLs in the database.
6. Log all actions, errors, and progress for debugging and monitoring purposes.